Projection system

ABSTRACT

A projection system includes a light source device, an imaging unit, and a projector. The light source device includes a projection surface including a reference area, and emits light including non-visible light and visible light from the reference area. The imaging unit receives non-visible light and captures an image of the projection surface. The projector projects a projection image of visible light on the projection surface based on the captured image captured by the imaging unit.

BACKGROUND 1. Field

The present disclosure relates to a projection system that projects animage onto an object.

2. Description of Related Art

Unexamined Japanese Patent Publication No. H09-24053 discloses a surgerysupporting system that outputs image data indicating an affected part ofa living body, which will undergo surgery, from a fluorescent imagingdevice, and reproduces an image based on the image data and displays theimage on the actual affected part by an image projection apparatus. Asubstance that emits fluorescence by irradiation of light having apredetermined wavelength is administered in advance to the affected partof the living body. That is, this system supports for confirmation of alesion by displaying, on the actual affected part, a fluorescent imageof the affected part emitting the fluorescence.

SUMMARY

The present disclosure provides a projection system that captures animage of a subject and projects a projection image, the projectionsystem enabling easy adjustment of a shift between the projection imageand the subject.

The projection system according to the present disclosure includes alight source device, an imaging unit, and a projector. The light sourcedevice has a projection surface including a reference area, and emitslight including non-visible light and visible light from the referencearea. The imaging unit receives non-visible light and captures an imageof the projection surface. The projector projects a projection image ofvisible light on the projection surface based on the captured imagecaptured by the imaging unit.

According to the present disclosure, the projection system, whichcaptures an image of a subject and projects a projection image, caneasily adjust a shift between the projection image and the subject.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram illustrating a configuration of surgerysupporting system 100.

FIG. 2A is a diagram illustrating the state of a surgical field before aprojection operation is performed in the surgery supporting system 100.

FIG. 2B is a diagram illustrating the state in which the projectionoperation is performed on the surgical field in FIG. 2A.

FIG. 3 is a schematic diagram illustrating a configuration of a shiftadjustment system 500.

FIG. 4A is a perspective view illustrating an appearance of opticaladjustment device 400.

FIG. 4B is an exploded perspective view illustrating the configurationof optical adjustment device 400.

FIG. 5A is a perspective view of optical adjustment device 400 duringshift adjustment.

FIG. 5B is a view illustrating one example of a state of projectionsurface 402 when shift adjustment has not yet been performed.

FIG. 5C is a view illustrating an image for projection in the example ofFIG. 5B.

FIG. 5D is a view illustrating an image for projection obtained byperforming the shift adjustment on the image in FIG. 5C.

FIG. 5E is a view illustrating one example of a state of projectionsurface 402 after the shift adjustment.

FIG. 6 is a view illustrating the state of projection surface 402 in anexample of use of optical adjustment device 400.

FIG. 7A is a plan view of opening mask 430′ in an application example.

FIG. 7B is a view illustrating a state in which an image is projected ona projection surface using opening mask 430′ in FIG. 7A.

FIG. 7C is a view illustrating the state in which the projectionoperation of surgery supporting system 100 is performed on theprojection surface illustrated in FIG. 7B.

FIG. 8A is a diagram for describing infrared fluorescence 310 andvisible laser light 320 before the shift adjustment.

FIG. 8B is a diagram for describing infrared fluorescence 310 andvisible laser light 320 after the shift adjustment.

FIG. 9 is a diagram for describing scanning patterns by projector 220.

FIG. 10 is a flowchart illustrating the projection operation of acutting aid line according to the detection of an affected part.

FIG. 11A is a diagram for describing the projection operation of acutting aid line with a first cutting allowable range.

FIG. 11B is a diagram for describing the projection operation of acutting aid line with a second cutting allowable range.

FIG. 12A is a view illustrating a state of conventional surgery.

FIG. 12B is a view for describing projection of surgery aid informationonto the surrounding of an affected part.

FIG. 13A is a top view of an auxiliary screen material on which surgeryaid information is not projected.

FIG. 13B is a top view of an auxiliary screen material on which surgeryaid information is projected.

FIG. 14 is a flowchart illustrating the process flow in a monitoringoperation of a use height.

FIG. 15A is a view for describing the monitoring operation when a useheight falls within an allowable range.

FIG. 15B is a view for describing the monitoring operation when a useheight falls outside an allowable range.

FIG. 16 is a timing chart for describing operations of an infraredexcitation light source, a TOF sensor, and a visible-light laser.

DETAILED DESCRIPTION

Exemplary embodiments will be described below in detail with referenceto the drawings as necessary. However, more than necessary detaileddescriptions will sometimes be omitted. For example, detaileddescriptions for matters which have already been well known in the artand redundant descriptions for substantially the same configurationswill sometimes be omitted. This is to prevent the following descriptionfrom becoming unnecessarily redundant to facilitate understanding of aperson skilled in the art.

Note that the accompanying drawings and the following description areprovided by the applicant in order for a person skilled in the art tosufficiently understand the present disclosure, and they are notintended to limit the subject matter set forth in the claims.

First Exemplary Embodiment 1. Outline of Surgery Supporting System

The outline of a surgery supporting system according to a firstexemplary embodiment will be described with reference to FIG. 1 as oneexample of the projection system according to the present disclosure.FIG. 1 is a schematic diagram illustrating a configuration of surgerysupporting system 100 according to the first exemplary embodiment.

Surgery supporting system 100 is a system that visually supports surgeryperformed on a patient by a doctor or the like in a surgery room or thelike using a projection image. When surgery supporting system 100 isused, a light-sensitive substance is administered into blood or the likeof patient 130 who undergoes surgery.

The light-sensitive substance emits fluorescence in response toexcitation light. The first exemplary embodiment describes the case inwhich indocyanine green (hereinafter referred to as “ICG”) is used asone example of the light-sensitive substance. The ICG is a reagentmedically approved and usable for a human body. The ICG emits infraredfluorescence having a wavelength of around 850 nm which is the peakwavelength when irradiated with infrared excitation light having awavelength of around 800 nm. When administered into blood, the ICGaccumulates on affected part 140 where blood or lymph is stagnant.Therefore, the area of affected part 140 can be specified by detectingan infrared fluorescence area emitting infrared fluorescence.

In this case, since the infrared fluorescence emitted from the area ofaffected part 140 is non-visible, a doctor or the like cannot directlyspecify the area of affected part 140 even by visually observingsurgical field 135. In view of this, surgery supporting system 100firstly detects an area of ICG emitting infrared fluorescence to specifythe area of affected part 140. Then, surgery supporting system 100 emitsvisible light to the specified area of affected part 140 in order toenable the specified area of affected part 140 to be visuallyrecognizable by a human. Thus, a projection image that enables thespecified area of affected part 140 to be visible is projected, wherebysurgery supporting system 100 can support identification of the area ofaffected part 140 by the doctor or the like who performs surgery.

2. Configuration of Surgery Supporting System

The configuration of surgery supporting system 100 will be describedbelow with reference to FIG. 1. Surgery supporting system 100 isinstalled and used in a surgery room in a hospital. Surgery supportingsystem 100 includes imaging irradiation device 200, control device 230,memory 240, and infrared excitation light source 250. Although notillustrated, surgery supporting system 100 also includes a mechanism forchanging the position where imaging irradiation device 200 is disposed.This mechanism includes, for example, a drive arm mechanically connectedto imaging irradiation device 200 or a caster for a pedestal on which aset of surgery supporting system 100 is placed.

Imaging irradiation device 200 integrally includes an imaging unit andan irradiation unit. Imaging irradiation device 200 includes infraredcamera 210, dichroic mirror 211, projector 220, and TOF (Time-of-Flight)sensor 260. Projector 220 includes visible-light laser 222 and MEMS(Micro Electro Mechanical System) mirror 221.

Control device 230 provided in a controller generally controls eachcomponent of surgery supporting system 100. Control device 230 iselectrically connected to infrared camera 210, visible-light laser 222,MEMS mirror 221, TOF sensor 260, memory 240, and infrared excitationlight source 250 and outputs a control signal for controlling eachcomponent. Control device 230 includes a CPU or an MPU, for example, andimplements its function by executing a predetermined program. Notably,the function of control device 230 may be implemented by an exclusivelydesigned electronic circuit or a reconfigurable electronic circuit(ASIC, FPGA, etc.).

Memory 240 includes a ROM (Read Only Memory) or a RAM (Random AccessMemory), for example. Memory 240 is a recording medium accessed bycontrol device 230 as necessary when control device 230 executes variouscalculations.

Infrared excitation light source 250 emits infrared excitation light 300with a spectrum including at least a wavelength range component around800 nm which is the excitation wavelength of the ICG. Infraredexcitation light source 250 can switch the on/off of the irradiation ofinfrared excitation light 300 according to the control signal fromcontrol device 230. In the example illustrated in FIG. 1, infraredexcitation light source 250 is disposed outside imaging irradiationdevice 200. However, it is not limited thereto. That is, infraredexcitation light source 250 may be disposed inside imaging irradiationdevice 200, if an irradiation opening for infrared excitation light isappropriately formed.

Next, the configuration of each component composing imaging irradiationdevice 200 will be described.

Infrared camera 210 used for an imaging unit is a camera that hasspectral sensitivity characteristics with high light receptionsensitivity in an infrared region. Surgery supporting system 100according to the present exemplary embodiment needs to detect infraredfluorescence having a wavelength of around 850 nm from the ICG. Forthis, infrared camera 210 having spectral sensitivity characteristicswith high light reception sensitivity for at least an infrared regionwith a wavelength around 850 nm is used. It is to be noted that, toprevent light other than the infrared fluorescence from the ICG frombeing received, a band pass filter that only allows passage of lighthaving a wavelength of about 850 nm may be provided in front of theimaging surface of infrared camera 210. The wavelength spectrum of theinfrared fluorescence is one example of a first spectrum. Infraredcamera 210 transmits a captured image (infrared image) indicating thecapturing result to control device 230.

Visible-light laser 222 is a laser device that emits visible light inprojector 220. A laser light source with an arbitrary wavelength may beused for visible-light laser 222, so long as it emits light in a visiblelight region visually recognizable by a human. Visible-light laser 222may include a laser light source of one color, or may be configured suchthat laser light sources of multiple colors may be switchable accordingto a control signal from control device 230. Visible-light laser 222emits visible laser light 320 to MEMS mirror 221.

MEMS mirror 221 has a lot of micro-mirror surfaces arranged on a plane,and includes a digital mirror device, for example. Visible laser light320 emitted from visible-light laser 222 is incident on each of themicro-mirror surfaces of MEMS mirror 221. MEMS mirror 221 reflectsvisible-light laser 222 in the direction according to the tilt angle ofeach of the micro-mirror surfaces, thereby generating a projection imageof visible light.

Here, control device 230 controls the tilt angle of each of themicro-mirror surfaces of MEMS mirror 221 horizontally and vertically.Thus, control device 230 can two-dimensionally scan visible laser light320 in the vertical direction and in the horizontal direction, therebybeing capable of allowing projector 220 to generate a projection image.Visible laser light 320 reflected on the micro-mirror surfaces of MEMSmirror 221 reaches dichroic mirror 211.

Although the present exemplary embodiment illustrates MEMS mirror 221 asone example of the component of projector 220, it is not limitedthereto. For example, a galvano mirror may be used. That is, anyarbitrary optical element can be used, so long as it enables scanning inthe horizontal direction and scanning in the vertical direction.

Dichroic mirror 211 is disposed to face each of infrared camera 210 andMEMS mirror 221. Dichroic mirror 211 is an optical element having afunction of transmitting light having a specific wavelength rangecomponent (including a wavelength of 850 nm) in incident light andreflecting light having other wavelength range components (includingvisible-light component). In the present exemplary embodiment, MEMSmirror 221 is disposed in the horizontal direction of dichroic mirror211, and infrared camera 210 is disposed above dichroic mirror 211 inthe vertical direction, as illustrated in FIG. 1. Due to the aboveoptical characteristic, dichroic mirror 211 reflects visible laser light320 emitted from visible-light laser 222 but transmits infraredfluorescence 310 directed to the imaging surface of infrared camera 210.

Also, as illustrated in FIG. 1, dichroic mirror 211, projector 220, andinfrared camera 210 are positioned such that the optical path of visiblelaser light 320 reflected by dichroic mirror 211 and the optical path ofinfrared fluorescence 310 incident on the imaging surface of infraredcamera 210 coincide with each other. Thus, the precision in emittingvisible laser light 320 to the area (affected part 140) emittinginfrared fluorescence 310 can be enhanced.

TOF sensor 260 detects distance information indicating a distance fromitself to an object by radiating infrared detection light 330 andreceiving infrared detection light 330 reflected on the object. Thewavelength spectrum of infrared detection light 330 is one example of asecond spectrum. TOF sensor 260 uses infrared light with a wavelength of850 nm to 950 nm as infrared detection light 330. The second spectrumcan be at least partly superimposed on the first spectrum. TOF sensor260 measures the distance from itself to the object on the basis of alag time from the radiation of infrared detection light 330 till thereception of infrared detection light 330 reflected on the object andlight speed. Alternatively, TOF sensor 260 may measure the distance fromitself to the object on the basis of the difference between the voltagevalue of infrared detection light 330 when emitted and the voltage valueof infrared detection light 330 when received after being reflected onthe object. TOF sensor 260 transmits the distance information concerningthe measured distance from itself to the object to control device 230.

Also, as illustrated in FIG. 1, surgical bed 110, shadowless lamp 120,and the like are installed in the surgery room in addition to surgerysupporting system 100. Surgical bed 110 is a table on which patient 130is laid. Shadowless lamp 120 is an illumination tool that illuminatesaffected part 140 of patient 130 lying on surgical bed 110. Shadowlesslamp 120 emits light having high illuminance (30,000 lux to 100,000 lux)for preventing the work area of the doctor from being shadowed.

Surgery supporting system 100 is placed such that imaging irradiationdevice 200 is located vertically above patient 130 lying on surgical bed110. In surgery supporting system 100 according to the present exemplaryembodiment, an allowable range of a use height is specified on the basisof the focal length determined by the optical system in infrared camera210 to ensure the precision in specifying the area of affected part 140by infrared camera 210. In the present exemplary embodiment, the heightof 1000 mm±300 mm from the body axis of patient 130 lying on surgicalbed 110 to imaging irradiation device 200 (TOF sensor 260) is specifiedas the allowable range of a use height. The allowable range of heightwill be described in detail below.

3. Basic Operation of Surgery Supporting System

Next, the activation operation and projection operation, which are thebasic operation of surgery supporting system 100, will be described.

3-1. Activation Operation of Surgery Supporting System

Firstly, the activation operation of surgery supporting system 100 willbe described. When a power source (not illustrated) is switched to onfrom off in surgery supporting system 100, control device 230 isactivated. Activated control device 230 executes the activationoperation of the components composing surgery supporting system 100,such as infrared camera 210, visible-light laser 222, infraredexcitation light source 250, and TOF sensor 260.

After the activation operation is executed, visible-light laser 222starts an amplifying operation of visible laser light 320. Imagingirradiation device 200 is usable at the timing at which the output ofvisible laser light 320 is stabilized.

3-2. Basic Projection Operation of Surgery Supporting System

Next, the basic projection operation of surgery supporting system 100will be described with reference to FIGS. 1, 2A, and 2B. FIGS. 2A and 2Bare diagrams illustrating the state of surgical field 135 in surgerysupporting system 100 in FIG. 1. FIG. 2A is a diagram illustrating thestate of surgical field 135 before the projection operation is performedin surgery supporting system 100. FIG. 2B is a diagram illustrating thestate in which the projection operation is performed on surgical field135 in FIG. 2A.

In the state illustrated in FIG. 2A, control device 230 firstly drivesinfrared excitation light source 250 to emit infrared excitation light300 to surgical field 135 including affected part 140. Then, infraredexcitation light 300 excites the ICG accumulated on affected part 140 insurgical field 135, so that affected part 140 emits infraredfluorescence 310.

Next, infrared camera 210 captures an image of affected part 140 insurgical field 135 under the control of control device 230. At thattime, the captured image includes an image of infrared fluorescence areaR310 from which infrared fluorescence 310 is emitted. Infrared camera210 transmits the captured image to control device 230.

Control device 230 detects infrared fluorescence area R310 on the basisof the captured image transmitted from infrared camera 210.Specifically, control device 230 calculates an XY coordinate from avertex of the captured image to acquire information indicating thecoordinate of infrared fluorescence area R310 in the captured image.

Memory 240 stores information indicating the correspondence relationbetween a coordinate in the captured image from infrared camera 210 anda coordinate in the data for generating a projection image with MEMSmirror 221. Control device 230 controls MEMS mirror 221 such thatvisible laser light 320 is emitted to the coordinate corresponding tothe acquired coordinate on the basis of the information indicating thecorrespondence relation stored in the storage unit, i.e., memory 240.Notably, projector 220 is controlled to scan and emit visible laserlight 320.

When visible laser light 320 is emitted, as illustrated in FIG. 2B,projection image G320 due to visible laser light 320 is projected ontoinfrared fluorescence area R310 in surgical field 135. In this way, insurgery supporting system 100, infrared fluorescence area R310 isdetected on the basis of the captured image by infrared camera 210,whereby the area of affected part 140 emitting invisible infraredfluorescence 310 is specified. Further, the area of affected part 140which is not directly visually recognizable can be made visible in thesurgical field due to the appropriate projection of projection imageG320 by projector 220. Notably, projection image G320 is a monochromeuniform image by visible-light laser 222, for example.

The process described above is repeatedly executed in a predeterminedcycle (for example, 1/60 second). Thus, a captured image is projectedonce per 1/60 second, for example, whereby a doctor or the like canvisually recognize the position and shape of affected part 140 in realtime.

4. Method for Adjusting Projection Shift in Surgery Supporting System

4-1. Outline of Method for Adjusting Projection Shift

As described above, surgery supporting system 100 detects affected part140 which is not visually recognizable and emits infrared fluorescence310 from ICG, using infrared camera 210 (see FIG. 2A), and projects aprojection image due to visible laser light 320 to make affected part140 visible using projection image G320 (see FIG. 2B). If projectionimage G320 is projected as being shifted from infrared fluorescence areaR310 of affected part 140 while surgery supporting system 100 is used,the position or the like of affected part 140 may be falsely recognizedin surgical field 135. Therefore, before surgery supporting system 100is used, the relation between the position specified on the basis of thecaptured image of infrared camera 210 and the projection position of theprojection image is confirmed, and if there is a positional shift,surgery supporting system 100 needs to be adjusted.

The confirmation of positional shift and adjustment of positional shiftare performed in various situations before surgery supporting system 100is used. For example, the adjustment of positional shift is performedwhen the arrangement in imaging irradiation device 200 is determined soas to allow visible-light laser 222 to emit visible laser light 320 tothe area specified by infrared camera 210 in a production step. Inaddition, the adjustment is performed also in the assembling step ofimaging irradiation device 200, since a very small error may begenerated between the irradiation position of visible-light laser 222and the imaging position of the infrared camera. Further, disturbanceafter the assembly and a difference in the angle of view betweeninfrared camera 210 and projector 220 also cause a positional shift.Since ensuring the safety is important in medical application, whetheror not there is a positional shift needs to be confirmed every timebefore the start of surgery using surgery supporting system 100.

According to the present invention, a target to be captured by infraredcamera 210 is easily made visible, and a positional shift of aprojection image can be easily visually recognized. Due to the methodfor adjusting a positional shift using an optical adjustment device, theshift between the irradiation position of visible-light laser 222 andthe imaging position of infrared camera 210 can easily be adjusted.

The configuration of the optical adjustment device and the method foradjusting a shift using the optical adjustment device will besequentially described.

4-2. Configurations of Shift Adjustment System and Optical AdjustmentDevice

The configuration of optical adjustment device will be described withreference to FIGS. 3, 4A, and 4B. FIG. 3 is a schematic diagramillustrating the configuration of shift adjustment system 500 thatadjusts a shift between the irradiation position of visible-light laser222 and the imaging position of infrared camera 210. FIGS. 4A and 4B areviews for describing the configuration of optical adjustment device 400.FIG. 4A is a perspective view illustrating the appearance of opticaladjustment device 400. FIG. 4B is an exploded perspective viewillustrating the configuration of optical adjustment device 400.

Shift adjustment system 500 includes surgery supporting system 100 andoptical adjustment device (light source device) 400. Shift adjustmentsystem 500 is one example of a projection system. FIG. 3 illustrates thearrangement state of optical adjustment device 400 with respect tosurgery supporting system 100 in shift adjustment system 500. Asillustrated in FIG. 3, optical adjustment device 400 has projectionsurface 402, which is a target of the imaging and projection operationsof surgery supporting system 100, on one surface of box-like housing401, and includes a light source in housing 401. As illustrated in FIG.4A, projection surface 402 of optical adjustment device 400 is also anemission surface of LED (Light Emitting Diode) light 340 emitted fromthe inside of housing 401. FIG. 4B illustrates the internal structure ofhousing 401 of optical adjustment device 400. As illustrated in FIG. 4B,optical adjustment device 400 includes white LED 410, diffusion plate420, opening mask 430, screen material 440, and protection glass 450.Optical adjustment device 400 has the structure in which white LED 410,diffusion plate 420, opening mask 430, screen material 440, andprotection glass 450 are stacked in this order in housing 401.

White LED 410 is a semiconductor light-emitting element that emits whiteLED light 340. The wavelength spectrum of light emitted from white LED410 includes a non-visible light region (including an infrared region)as well as a visible light region. In the present disclosure, white LED410 is used as the light source of optical adjustment device 400.However, it is not limited thereto. In place of white LED 410, a lightsource having a spectrum including a visible light component and anon-visible light component (including an infrared wavelength component)may be used. For example, both of a light-emitting element that emitsonly visible light, such as a monochrome LED, and a light-emittingelement that emits only infrared light may be disposed in housing 401 toconstitute a light source. Alternatively, an arbitrary light source thatcan coaxially emit visible light and infrared light may be used.

Diffusion plate 420 is made of a resin plate having a rough groundedglass surface, for example. Diffusion plate 420 is disposed to facewhite LED 410 in housing 401. Diffusion plate 420 reduces brightnessunevenness of light emitted from white LED 410 and emits the resultantlight from surfaces. Notably, optical adjustment device 400 may notinclude diffusion plate 420.

Opening mask 430 is a light-shielding member having opening 460 formedon light-shielding surface 470. Opening mask 430 is disposed to facewhite LED 410 through diffusion plate 420 in housing 401 of opticaladjustment device 400. Opening 460 is a hole facing white LED 410 andhaving a predetermined size, and light emitted from white LED 410 passesthrough opening 460. Light-shielding surface 470 encloses opening 460 toshield light incident from white LED 410. The size of opening 460 or thelocation on light-shielding surface 470 of opening mask 430 isdetermined according to the purpose of measurement. For example, opening460 with a size of 2 mm or less is formed on opening mask 430 to confirmwhether or not a shift is 2 mm or less.

Screen material 440 is a sheet-like member having light-scatteringproperty, and has projection surface 402 on one main surface. Screenmaterial 440 is disposed to face opening mask 430 with its main surface,which is not projection surface 402, facing opening mask 430. At least avisible-light component of light emitted from white LED 410 is scatteredon screen material 440. Thus, a viewing angle of reference area Ra,which is an area irradiated with light emitted from white LED 410 andradiating this light, is increased as illustrated in FIG. 4A, andtherefore, reference area Ra can be easily visually recognized by ahuman. Reference area Ra irradiated with light from white LED 410 isformed to have a size according to the setting of opening 460, andserves as a reference for visually recognizing a positional shift in theshift adjustment method described below.

The material of screen material 440 is paper, for example. The color ofpaper is arbitrary, and a color (for example, complementary color) whichfacilitates visual recognition according to the color of emitted laserlight may be used. Alternatively, cloth may be used for the material ofscreen material 440 instead of paper. An arbitrary material thatscatters at least a part of visible-light component of incident lightand has small scattering rate of an infrared wavelength component may beused as the material of screen material 440.

Protection glass 450 is a glass member that protects screen material 440from having scratches. Notably, optical adjustment device 400 may notinclude screen material 440 and protection glass 450.

4-3. Shift Adjustment Method Using Optical Adjustment Device

Next, the shift adjustment method using optical adjustment device 400will be described with reference to FIGS. 3 and 5A to 5E. FIGS. 5A to 5Eare views for describing shift adjustment using optical adjustmentdevice 400. FIG. 5A is a perspective view of the optical adjustmentdevice 400 during shift adjustment. FIG. 5B is a view illustrating oneexample of a state of projection surface 402 when the shift adjustmenthas not yet been performed. FIG. 5C is a view illustrating an image forprojection in the example of FIG. 5B. FIG. 5D is a view illustrating animage for projection obtained by performing the shift adjustment on theimage in FIG. 5C. FIG. 5E is a view illustrating one example of a stateof projection surface 402 after the shift adjustment.

The present adjustment method is performed by, for example, an operatorfor adjustment of a manufacturer as an adjustment operation in aproduction step of imaging irradiation device 200 or surgery supportingsystem 100. In this case, shipped items of imaging irradiation device200 or surgery supporting system 100 have already been adjusted. Thepresent adjustment method can also be performed as a precautionaryconfirmation operation just before actual surgery, even if adjustmenthas already been performed in the production step.

When performing the present adjustment method, an operator foradjustment places optical adjustment device 400 at the position justbelow imaging irradiation device 200 and facing the imaging surface ofinfrared camera 210 and irradiation opening of visible laser light 320as illustrated in FIG. 3. At that time, if the height allowable range,which is an allowable range of the distance (height) between imagingirradiation device 200 and surgical bed 110, is set as 1000 mm±300 mm,optical adjustment device 400 is placed at the position 1000 mm from thelower surface of imaging irradiation device 200.

After placing optical adjustment device 400, the operator for adjustmentallows white LED 410 to emit LED light 340.

LED light 340 is incident on screen material 440 through opening mask430, and emitted from reference area Ra on projection surface 402. Thevisible-light component of LED light 340 generates scattering light onscreen material 440. The scattering light of the visible-light componentof LED light 340 forms an image (hereinafter referred to as “referencearea image” Ra) indicating reference area Ra on projection surface 402(see FIGS. 4A and 4B).

LED light 340 emitted from white LED 410 includes a wavelength rangecomponent of an infrared region. The wavelength range component for theinfrared region in LED light 340 passes through dichroic mirror 211 ofsurgery supporting system 100.

Surgery supporting system 100 performs the projection operationdescribed above for projection surface 402 of optical adjustment device400 as the imaging and projection target. In surgery supporting system100, infrared camera 210 receives light passing through dichroic mirror211 and captures an image of projection surface 402. Therefore, infraredcamera 210 captures an image of reference area Ra that emits lightincluding the wavelength range component of the infrared region.Infrared camera 210 transmits the captured image to control device(adjustment unit) 230.

Control device 230 calculates, for example, XY coordinate from onevertex of the captured image to acquire information indicating thecoordinate of reference area image Ra emitting light with the wavelengthrange of the infrared region, on the basis of the captured imagetransmitted from infrared camera 210. Control device 230 manages thecoordinate on the captured image transmitted from infrared camera 210and the scanning coordinate to which visible laser light 320 is to beemitted in one-to-one correspondence on image data, for example. Controldevice 230 controls MEMS mirror 221 such that visible laser light isemitted to the scanning coordinate corresponding to the acquiredcoordinate.

Projector 220 emits visible laser light 320 to optical adjustment device400 according to infrared emission from optical adjustment device 400,thereby projecting projection image Rb on projection surface 402 asillustrated in FIG. 5A. As a result, reference area image Ra that is theimaging target of imaging irradiation device 200 and projection image Rbby imaging irradiation device 200 are projected on projection surface402 of optical adjustment device 400 by visible light, and the operatorfor adjustment can visually recognize both images simultaneously.

At that time, reference area image Ra by LED light 340 and projectionimage Rb by visible laser light 320 have to coincide with each other.However, a positional shift may actually occur between both images dueto an assembling error or the like. In such a case, positional shifts Δxand Δy between the position of reference area image Ra and the positionof projection image Rb can be visually recognized by means of opticaladjustment device 400 as illustrated in FIG. 5B.

The adjustment operation for positional shifts Δx and Δy illustrated inFIG. 5B will be described below.

Firstly, control device 230 stores, in memory 240, informationindicating the irradiation position (that is, the scanning position ofMEMS mirror 221) of visible laser light 320 when the shift adjustmenthas not yet been performed. In this case, control device 230 generatesan image signal indicating image Db in which projection image Rb1 isformed based on the capturing result of reference area image Ra asillustrated in FIG. 5C. Projection image Rb by visible laser light 320is projected on projection surface 402 on the basis of this image signalat the position shifted from reference area image Ra as illustrated inFIG. 5B. Control device 230 stores, in memory 240, position P1 ofprojection image Rb1 on image Db while the shift adjustment has not yetbeen performed. Position (scanning position) P1 is referred to as an“unadjusted position” below.

The operator for adjustment compares reference area image Ra withprojection image Rb projected on projection surface 402 while observingboth images, and inputs a shift amount to control device 230 using anoperation unit (not illustrated) or the like so as to align both images.Specifically, the operator for adjustment inputs, to control device 230,information concerning an amount of movement for shifting the projectionimage on an X axis or a Y axis.

Control device 230 controls projector 220 such that the irradiationposition (scanning position by MEMS mirror 221) of visible laser light320 is changed on the basis of the input information. For example, onthe basis of the input information indicating the amount of movement,control device 230 shifts the irradiation position on image Db fromunadjusted position P1 by amounts of movement Δxd and Δyd indicated bythe input information as illustrated in FIG. 5D. Amounts of movement Δxdand Δyd on the image are values corresponding to actual positional shiftamounts Δx and Δy on projection surface 402. Due to this adjustment,projection image Rb2 is projected on the position on projection surface402 corresponding to irradiation position P2 after the adjustment, andaligned with reference area image Ra as illustrated in FIG. 5E.

The operation described above is repeated until the operator foradjustment determines that reference area image Ra and projection imageRb2 projected on projection surface 402 are aligned with each other.

Upon the completion of the adjustment operation, control device 230stores the last irradiation position P2 (that is, the scanning positionof MEMS mirror 221) on image Db in memory 240. Irradiation position(scanning position) P2 is referred to as an “adjusted position” below.

Control device 230 calculates a shift correction amount on the basis ofunadjusted position P1 and adjusted position P2 stored in memory 240.Specifically, control device 230 calculates the difference betweenunadjusted position P1 and adjusted position P2 as a shift correctionamount. In the example illustrated in FIGS. 5B to 5E, amounts ofmovement Δxd and Δyd are stored in memory 240 as the shift correctionamount.

After performing the shift adjustment described above, control device230 corrects the irradiation position of visible laser light 320 on thebasis of the shift correction amount stored in memory 240, and projectsa projection image. Thus, a projection image can precisely be projectedon a projection target.

4-4. Application Example of Optical Adjustment Device 400

Whether or not a positional shift amount falls within an allowable errorcan be confirmed from a reference area image and a projection imageprojected on optical adjustment device 400. The confirmation method ofan allowable error will be described below with reference to FIG. 6.FIG. 6 illustrates one example of a state of projection surface 402 whenoptical adjustment device 400 is used while being disposed asillustrated in FIG. 3.

Diameter La of circular reference area image Ra illustrated in FIG. 6 isset to be equal to a predetermined allowable error according to thespecification of surgery supporting system 100. Diameter La is setaccording to the size of opening 460 of opening mask 430 (see FIGS. 4Aand 4B). For example, when the specification of surgery supportingsystem 100 requires projection precision of an allowable error of 2 mm,diameter La is set to be 2 mm. In one example illustrated in FIG. 6, itis supposed that there is no error in projection magnification ofprojection image Rb.

In the case where reference area image Ra and projection image Rb arepartly overlapped with each other as illustrated in FIG. 6, positionalshift ΔL between reference area image Ra and projection image Rb becomesequal to or less than diameter La. Therefore, the projection precisionof surgery supporting system 100 falls within the allowable error range.On the other hand, when there is no overlapped portion between referencearea image Ra and projection image Rb, positional shift ΔL is largerthan diameter La, so that the projection precision can be determined tobe outside the allowable error range. Accordingly, a user of opticaladjustment device 400 can easily confirm whether or not the positionalshift falls within the allowable error range by visually recognizingwhether or not reference area image Ra and projection image Rb are atleast partly overlapped with each other. If the positional shift isconfirmed to fall within the allowable error range, the operation ofcontrol device 230 for the above-mentioned shift adjustment may not beperformed.

Further, in the above description, the shape of reference area image Rais circular. However, the shape of reference area image Ra is notparticularly limited, and reference area image Ra may have an ellipticshape, a polygonal shape such as a triangular shape or a rectangularshape, or other shape. In addition, a plurality of reference areas maybe formed on one projection surface 402. As one example, a shiftadjustment method in the case where the reference area image isrectangle will be described with reference to FIGS. 7A to 7C.

FIG. 7A is a plan view of opening mask 430′. FIG. 7B illustrates thestate in which an emission image by white LED 410 is projected onprojection surface 402 using opening mask 430′. FIG. 7C illustrates thestate in which the projection operation of surgery supporting system 100disposed as illustrated in FIG. 3 is performed on projection surface 402illustrated in FIG. 7B.

As illustrated in FIG. 7A, with the use of opening mask 430′ havingrectangular opening 460′, rectangular reference area image Ra′ isprojected on projection surface 402 as illustrated in FIG. 7B. In thiscase, whether or not the orientation of reference area image Ra′ and theorientation of projection image Rb′ are different from each other can bevisually confirmed. For example, angular shift Δθ can be visuallyrecognized through comparison between one vertex of reference area imageRa′ and one vertex of projection image Rb′ as illustrated in FIG. 7C.Therefore, the operator for adjustment can perform adjustment whilevisually recognizing angular shift Δθ, as in the adjustment forpositional shifts Δx and Δy.

4-5. Effects, Etc.

As described above, in the present exemplary embodiment, shiftadjustment system 500 includes optical adjustment device 400, infraredcamera 210, and projector 220. Optical adjustment device 400 hasprojection surface 402 including reference area Ra, and emits LED light340 including non-visible light and visible light from reference areaRa. Infrared camera 210 receives non-visible light and captures an imageof projection surface 402. Projector 220 projects projection image Rb ofvisible light on projection surface 402 on the basis of the capturedimage captured by infrared camera 210.

Thus, LED light including visible light is emitted from reference areaRa included in projection surface 402 of optical adjustment device 400,and projection image Rb of visible light based on the captured image ofreference area Ra is projected on projection surface 402. Accordingly,the shift between reference area Ra, which is a subject, on projectionsurface 402 and its projection image Rb is made visible. Consequently,the projection system, which captures an image of the subject andprojects the projection image, can easily adjust the shift between thesubject and the projection image.

Further, the shift adjustment method is an adjustment method foradjusting projection image G320 which is to be projected on affectedpart 140 in surgery supporting system 100. Surgery supporting system 100includes infrared camera 210 that receives infrared fluorescence 310 andcaptures an image of affected part 140, and projector 220 that generatesprojection image G320 of visible light on the basis of the capturedimage of affected part 140 and projects projection image G320 onaffected part 140. The shift adjustment method includes a step ofemitting LED light 340 including a spectrum having a visible-lightcomponent and infrared wavelength component (including a wavelength of850 nm) to reference area Ra on projection surface 402 that is thetarget for imaging and projection operations of surgery supportingsystem 100. The shift adjustment method includes a step of capturingreference area Ra on projection surface 402 by infrared camera 210. Theshift adjustment method includes a step of projecting projection imageRb by projector 220 based on captured reference area Ra on projectionsurface 402 onto projection surface 402. The shift adjustment methodincludes a step of comparing reference area Ra with projection image Rbon projection surface 402. The shift adjustment method includes a stepof adjusting the position of projection image Rb on the basis of thecomparison result.

Further, in the present exemplary embodiment, optical adjustment device400 is an adjustment device for adjusting projection image G320 which isto be projected on affected part 140 in surgery supporting system 100.Optical adjustment device 400 includes white LED 410 and projectionsurface 402. White LED 410 emits LED light 340 having a spectrumincluding a visible-light component and an infrared wavelength component(including a wavelength of 850 nm). Projection surface 402 includespredetermined reference area Ra irradiated with (white) LED light 340emitted from white LED 410, and becomes a target for imaging andprojection operations of surgery supporting system 100.

Thus, while visible laser light 320 is emitted to the area from whichinfrared fluorescence 310 that is fluorescence of ICG is detected duringactual surgery, infrared light included in white LED 410 of opticaladjustment device 400 is regarded as infrared fluorescence of ICG duringthe adjustment operation. With this, the shift between the irradiationposition of visible-light laser 222 and the imaging position of infraredcamera 210 can be made visible on projection surface 402, and thus, theshift can easily be adjusted. Consequently, visible laser light 320 canaccurately be emitted to the area of affected part 140 detected andspecified by infrared camera 210.

Notably, while projection surface 402 is a main surface of screenmaterial 440 in the above description, it is not limited thereto. Forexample, in an optical adjustment device not having screen material 440,light-shielding surface 470 of opening mask 430 may be used as aprojection surface. In this case as well, a reference area to which LEDlight 340 is emitted is formed by opening 460.

In addition, while reference area Ra is formed by opening 460 in theabove description, it is not limited thereto. A reference area may beformed by guiding LED light 340 to be incident on projection surface 402using a reflection mirror or a lens.

Further, in the above description, a projection image is adjusted with asignal process based on a shift correction amount. However, the shiftadjustment method according to the present exemplary embodiment is notlimited thereto. For example, an operator for adjustment may adjust aphysical arrangement of infrared camera 210, visible-light laser 222, orthe like while visually observing projection surface 402 of opticaladjustment device 400.

In addition, while an operator for adjustment aligns reference area Raand projection image Rb with each other by operating an operation unit,it is not limited thereto. Control device 230 may compare reference areaRa with projection image Rb on projection surface 402, and adjust theposition of the projection image on the basis of the comparison result.The positions of reference area Ra and visible-light area Rb may bespecified by capturing projection surface 402 by a visible light camera,and control device 230 may perform position alignment. For example, anumber of dots on a captured image by the visible-light camera may becounted, and the counted number may be converted into a correctionamount. Control device 230 may execute such a process by using apredetermined program.

While the case in which Δxd and Δyd are stored in memory 240 as a shiftcorrection amount has been described above, the configuration is notlimited thereto. In the case where rotation angle θ of projection imageRb with respect to reference area Ra and projection magnification Z arechanged during alignment between reference area Ra and projection imageRb, correction amount Δθd of rotation angle θ and correction amount ΔZdof projection magnification Z may be stored in memory 240. Notably,projection magnification Z and its correction amount ΔZd may be setaccording to a zoom value with an optical system for projecting aprojection image, such as a zoom lens, or may be set according to adigital value in a signal process for a projection image.

For example, correction amount Δθd can be extracted on the basis ofangular shift Δθ illustrated in FIG. 7C. Further, correction amount ΔZdcan be extracted by comparison in distance between two vertexes betweenreference area Ra′ and projection image Rb′ illustrated in FIG. 7C. Inaddition, a visible-light camera may capture an image of opticaladjustment device 400 in various different arrangement, and referencearea image Ra and projection image Rb may be compared in each case toextract and correct distortion of the projection image.

Further, while a shift is adjusted by using one optical adjustmentdevice 400 in the above description, a plurality of optical adjustmentdevices 400 may be used for shift adjustment. With this, a shift can beadjusted without changing the location of optical adjustment device 400,whereby the adjustment time can be shortened and precision in adjustmentcan be enhanced.

Further, while the shift adjustment method for the case where projector220 includes visible-light laser 222 and scans and emits laser has beendescribed above, the projection method of a projection image is notlimited thereto. The shift adjustment method using optical adjustmentdevice 400 can also be applied for the case of projecting a projectionimage with other method.

5. Scanning Operation by Laser Scanning Projection

5-1. Outline of Scanning Operation

In surgery using surgery supporting system 100, lighting devices withhigh illuminance (30,000 lux to 100,000 lux) such as illumination fromshadowless lamp 120 and illumination attached to the head of a doctormay simultaneously be used in some cases. A light source used for anordinary imaging irradiation device 200 has low illuminance such asabout hundreds of lux, so that a projection image is inconspicuous andis not visually recognizable under an environment of high illuminance.

The present invention has devised the feature of employing a laserscanning projection using projector 220 which includes visible-lightlaser 222 and MEMS mirror 221 in surgery supporting system 100.Specifically, surgery supporting system 100 scans only the inside orboundary of an area of affected part 140, which is detected andspecified by infrared camera 210, with visible laser light 320 by MEMSmirror 221, while enabling supply of high-illumination light fromvisible-light laser 222. Thus, visual recognition of a projection imagecan be facilitated even under an environment of high illuminance, whilein consideration of safety.

5-2. Detail of Scanning Operation

The scanning operation with visible-light laser 222 and MEMS mirror 221will be described below with reference to FIGS. 1, 8A, 8B, and 9. FIGS.8A and 8B are diagrams for describing infrared fluorescence 310 andvisible laser light 320 before and after the shift adjustment. FIG. 9 isa diagram for describing scanning patterns with visible-light laser 222and MEMS mirror 221.

As illustrated in FIG. 1, surgical bed 110 on which patient 130 is laidis placed at the position just below imaging irradiation device 200 andfacing the imaging surface of infrared camera 210 and irradiationopening of visible laser light 320. In this case, if the allowable rangebased on the focal length of infrared camera 210 is set as 1000 mm±300mm, for example, the use height of imaging irradiation device 200 or theuse height of surgical bed 110 is adjusted such that the body axis ofpatient 130 is located at the position 1000 mm from the lower surface ofimaging irradiation device 200.

It is supposed that ICG has already been administered into blood ofpatient 130, and the ICG has accumulated on affected part 140. Patient130 lies on surgical bed 110 in the state in which a body part havingaffected part 140 which is to be cut with a scalpel faces upward, andwith this state, the operation of surgery supporting system 100 isstarted.

Firstly, control device 230 causes infrared excitation light source 250to emit infrared excitation light 300 having a wavelength around theexcitation wavelength of 800 nm of the ICG to surgical field 135 nearaffected part 140 of patient 130. The ICG accumulated on affected part140 induces an excitation reaction by infrared excitation light 300,thereby emitting infrared fluorescence 310 around a peak wavelength 850nm. A part of infrared fluorescence 310 emitted from the ICG accumulatedon affected part 140 passes through dichroic mirror 211. Infrared camera210 receives infrared fluorescence 310 passing through dichroic mirror211 and captures an image of surgical field 135. With this, the imagecaptured by infrared camera 210 includes infrared fluorescence area R310emitting infrared fluorescence 310. Infrared camera 210 transmits thecaptured image to control device 230.

Control device 230 specifies the coordinate (for example, XY coordinatefrom one vertex of the captured image) of the area emitting infraredfluorescence 310 on the basis of the captured image transmitted frominfrared camera 210. At that time, control device 230 reads shiftcorrection amounts Δx and Δy stored in memory 240. Control device 230also calculates a corrected coordinate obtained by correcting thespecified coordinate based on the captured image transmitted frominfrared camera 210 by the shift correction amounts read from memory240. Control device 230 controls MEMS mirror 221 such that visible laserlight 320 is emitted with a laser scanning pattern which is previouslyset to a scanning coordinate corresponding to the corrected coordinateof the coordinate in the captured image transmitted from infrared camera210. The detail of the laser scanning pattern will be described below.

FIG. 8A illustrates infrared fluorescence area R310 of infraredfluorescence 310 from ICG and projection area R320′ by visible laserlight 320 in the case where the correction based on the shift correctionamounts is not performed. When the corrected coordinate corrected by theshift correction amounts is not used, visible laser light 320 is emittedto the position shifted from infrared fluorescence area R310 of ICG byΔx and Δy.

On the other hand, FIG. 8B illustrates infrared fluorescence area R310of infrared fluorescence 310 from ICG and projection area R320 byvisible laser light 320 in the case where the correction based on theshift correction amounts is performed. When the corrected coordinatecorrected by the shift correction amounts is used, visible laser light320 is accurately emitted to infrared fluorescence area R310 of ICG.

As described above, use of the corrected coordinate enables accurateemission of visible laser light 320 to the area of affected part 140emitting infrared fluorescence 310.

Subsequently, the laser scanning pattern with visible-light laser 222and MEMS mirror 221 will be described. FIG. 9 illustrates rasterscanning and vector scanning that are selectable as the laser scanningpattern in surgery supporting system 100.

Raster scanning is a scanning pattern in which reciprocating emissionoperation of visible laser light 320 is performed on only the inside ofaffected part 140 emitting infrared fluorescence 310 so as to fill aface. During the raster scanning illustrated in FIG. 9, the scale ofilluminance is set to 1. Upon oscillation in 25 lumens, the illuminanceon the irradiated surface is about 2500 lux in the case where theirradiation area is the maximum (100 mm×100 mm), and is about 250,000lux in the case of the minimum irradiation area (10 mm×10 mm).

Vector scanning is a scanning pattern in which visible laser light 320is emitted to only the boundary of affected part 140 emitting infraredfluorescence 310 so as to draw a line. During the vector scanningillustrated in FIG. 9, the scale of illuminance is set to 20. Uponoscillation in 25 lumens, the illuminance on the irradiated surface isabout 50,000 lux in the case where the irradiation area is the maximum(100 mm×100 mm), and is about five million lux in the case of theminimum irradiation area (10 mm×10 mm).

A doctor can select which one of the visible-light laser irradiationwith the raster scanning and the visible-light laser irradiation withthe vector scanning is used by operating an operation unit (notillustrated) according to a surgery matter or the like.

Although FIG. 9 illustrates the raster scanning and the vector scanningas the scanning patterns, it is not limited thereto. For example, as aderived pattern of the raster scanning, a pattern may be used in whichonly the inside of the area of affected part 140 emitting infraredfluorescence 310 is scanned while thinned scanning is performed asnecessary. Alternatively, as a derived pattern of the raster scanning orthe vector scanning, a pattern in which the same site is continuouslyscanned more than once, and then, the irradiation position is shifted tothe other site may be used.

Control device 230 causes projector 220 to emit visible laser light 320to the area of affected part 140 emitting infrared fluorescence 310 onthe basis of the set scanning pattern so as to project a projectionimage. In this case, control device 230 controls MEMS mirror 221 suchthat visible-light laser is emitted on the basis of the set scanningpattern. Control device 230 continues the scanning operation even aftera round of scan to the inside or the boundary of the area of affectedpart 140 emitting infrared fluorescence 310 is completed.

5-3. Effects, Etc.

As described above, in the present exemplary embodiment, surgerysupporting system 100 includes infrared camera 210, projector 220, andcontrol device 230. Infrared camera 210 captures affected part 140.Projector 220 generates projection image G320 of visible light on thebasis of the captured image captured by infrared camera 210, andprojects projection image G320 onto affected part 140. Control device230 controls the operations of infrared camera 210 and projector 220.Projector 220 includes visible-light laser 222 that emits visible laserlight 320. Control device 230 controls projector 220 such thatprojection area R320 on which projection image G320 is projected isscanned with visible laser light 320 with a predetermined scanningpattern.

Since surgery supporting system 100 uses a laser light source havinghigh illuminance as an irradiation light source, visibility can beenhanced even under the environment with high illuminance by otherlighting devices such as shadowless lamp 120. Further, since only aninside or a boundary of a specific area is scanned with a predeterminedscanning pattern, illuminance can be obtained in comparison withirradiation to a wide area, whereby visibility can be enhanced. Further,surgery supporting system 100 is not configured to continuously emithigh-illuminance visible laser light 320 to the same position, but toscan an irradiation position. Thus, surgery supporting system 100 thatfacilitates visual recognition even under an environment of highilluminance, while taking into consideration of safety, can be provided.

The scanning pattern may be raster scanning in which visible laser light320 scans the inside of projection area R320. Alternatively, thescanning pattern may be vector scanning in which visible laser light 320scans along the boundary of projection area R320.

Projector 220 may further include MEMS mirror 221 having multiplemicro-mirror surfaces that reflect visible laser light 320. Controldevice 230 may control projector 220 such that visible laser light 320is scanned by changing the tilt angle of each of micro-mirror surfaceson MEMS mirror 221. Thus, the processing amount during the scan ofvisible laser light 320 can be reduced.

6. Projection Operation of Cutting Aid Line According to Detection ofAffected Part

6-1. Outline of Projection Operation of Cutting Aid Line

A doctor has to determine a cutting position which is to be cut with ascalpel before the start of surgery for affected part 140. Therefore,the doctor performs work for confirming the relation between affectedpart 140 and the cutting position which is to be cut with a scalpel onan image analysis device or the like. In this case, the doctor plans thecutting position so as to put the scalpel into affected part 140 with amargin of a certain distance. Then, the doctor prepares for surgery withthe planned cutting position in mind.

However, it is not easy to precisely reproduce the cutting positionplanned before the start of the surgery, and this imposes a strain onthe doctor. Further, the work described above takes much time forpreparation before the start of the surgery.

In view of this, the inventor of the present disclosure has conceived ofprojecting cutting aid line 321 for aiding determination of the cuttingposition which is to be cut with a scalpel as well as projectingprojection image G320 of visible light displaying the area of affectedpart 140 on which ICG is accumulated. According to this, thereproduction of the cutting position planned before the start of thesurgery can be assisted, whereby the burden of the doctor can bereduced. Further, the time for preparation before the start of thesurgery can be shortened.

6-2. Detail of Projection Operation of Cutting Aid Line

The projection operation of cutting aid line 321 according to thedetection of affected part 140 will be described with reference to FIGS.10, 11A, and 11B. FIG. 10 is a flowchart illustrating the projectionoperation of cutting aid line 321 according to the detection of affectedpart 140. FIGS. 11A and 11B are diagrams for describing the projectionoperation of cutting aid line 321 according to the detection of affectedpart 140.

It is supposed here that, before the start of surgery supported bysurgery supporting system 100, a doctor plans a cutting position whichis to be cut with a scalpel with allowance of a certain distance(hereinafter referred to as a “cutting allowable range”) to affectedpart 140. It is also supposed that the doctor inputs the planned cuttingallowable range to surgery supporting system 100 using an operation unit(not illustrated). For example, if the planned cutting allowable rangeis 2 centimeters, the doctor inputs information indicating the conditionfor the cutting aid line with the cutting allowable range of 2centimeters to surgery supporting system 100. Control device 230 insurgery supporting system 100 stores the cutting allowable range intomemory 240 on the basis of the input information.

The flowchart illustrated in FIG. 10 is started when the surgerysupported by surgery supporting system 100 is started with theinformation indicating the condition for the cutting aid line beingstored in memory 240.

Firstly, control device 230 reads the cutting allowable range or thelike stored in memory 240 to acquire the condition for the cutting aidline (S400).

Then, control device 230 causes infrared camera 210 to capture thefluorescent image of infrared fluorescence 310 emitted from ICG inresponse to infrared excitation light 300 (S401). At that time, controldevice 230 specifies the coordinate of the area emitting infraredfluorescence from the captured image transmitted from infrared camera210. Control device 230 also reads a shift correction amount from memory240, and calculates a corrected coordinate obtained by correcting thespecified coordinate based on the captured image transmitted frominfrared camera 210 by the shift correction amount. In this way, controldevice 230 detects infrared fluorescence area R310 in affected part 140.

Then, control device 230 starts the irradiation of visible laser light320 on the basis of the calculated corrected coordinate (S402). At thattime, control device 230 calculates the position where cutting aid line321 is to be projected on the basis of detected infrared fluorescencearea R310 and the cutting allowable range acquired in step S400. Then,control device 230 controls MEMS mirror 221 such that laser scanning isperformed on the area specified as affected part 140 and cutting aidline 321 is projected at the position away from the area specified asaffected part 140 by the cutting allowable range.

Further, in the process in step S402, control device 230 adjusts theprojection magnification on the basis of the distance informationdetected by TOF sensor 260. If the cutting allowable range is set as 2centimeters, control device 230 controls MEMS mirror 221 such thatcutting aid line 321 is projected at the position away from the areaspecified as affected part 140 by 2 centimeters. With this, cutting aidline 321 is projected at the position away from the area specified asaffected part 140 by 2 centimeters around the area with the shapesimilar to the area. The projection of cutting aid line 321 will bedescribed in more detail with reference to FIGS. 11A and 11B.

FIG. 11A illustrates surgical field 135 in the state in which theprojection operation of cutting aid line 321 according to the detectionof affected part 140 is performed in the case where first cuttingallowable range W1 is set. FIG. 11B illustrates surgical field 135 inthe state in which the projection operation of cutting aid line 321according to the detection of affected part 140 is performed in the casewhere second cutting allowable range W2 is set. Second cutting allowablerange W2 is supposed to be set larger than first cutting allowable rangeW1.

In FIGS. 11A and 11B, projection image G320 of visible light isprojected on infrared fluorescence area R310 in affected part 140emitting infrared fluorescence 310 in surgical field 135 according tothe detection of infrared fluorescence 310 in the captured image.Control device 230 sets the irradiation position of visible laser light320 for projecting cutting aid line 321 so as to enclose infraredfluorescence area R310 in surgical field 135 with a space of cuttingallowable ranges W1 and W2 on the basis of the distance informationdetected by TOF sensor 260, as well as the irradiation position ofprojection image G320. Therefore, as illustrated in FIGS. 11A and 11B,surgery supporting system 100 can change the position on which cuttingaid line 321 is projected according to the plan (cutting allowablerange) of the cutting position planned by the doctor.

Returning to FIG. 10, control device 230 repeats the processes in S401and S402 until the doctor or the like issues an end instruction throughthe operation unit (No in S403). When the end instruction is issued (Yesin S403), control device 230 ends the irradiation operation of visiblelaser light 320.

In the description of the flowchart in FIG. 10, the condition forcutting aid line 321 is cutting allowable ranges W1 and W2. However, thecondition for cutting aid line 321 is not limited thereto, and it may bea threshold in a distribution of intensity of infrared fluorescence 310,for example. In this case, control device 230 extracts the boundary ofthe intensity distribution in the captured image on the basis of thecaptured image captured by infrared camera 210 and the threshold set asthe condition for cutting aid line 321, and causes projector 220 toproject cutting aid line 321 on the extracted boundary in the process instep S402. With this, in the case where, for the removal of a part of anorgan (for example, one segment of a liver in Couinaud classificationsystem), ICG is administered with blood flow or the like being limitedso as to allow the portion to be removed to emit fluorescence, thedoctor or the like can visually recognize the cutting position for theremoval of the portion to be removed on the surface of the organ.

Further, as the condition for cutting aid line 321, both a threshold inthe intensity distribution of infrared fluorescence and cuttingallowable ranges W1 and W2 may be used, and cutting aid line 321 may beprojected at the position away from the boundary of the intensitydistribution in the captured image by cutting allowable ranges W1 andW2. Further, in the case where cutting aid line 321 is projected on theboundary of the intensity distribution of infrared fluorescence, controldevice 230 may determine the irradiation position through an imageanalysis of the captured image by infrared camera 210 withoutparticularly using the distance information of TOF sensor 260.

6-3. Effects, Etc.

As described above, in the present exemplary embodiment, surgerysupporting system 100 includes infrared camera 210, projector 220, andcontrol device 230. Infrared camera 210 captures an image of affectedpart 140. Projector 220 generates projection image G320 by visible lightand projects the resultant on affected part 140. Control device 230detects infrared fluorescence area R310 in affected part 140 emittinginfrared fluorescence 310 on the basis of the captured image captured byinfrared camera 210. Control device 230 causes projector 220 to projectprojection image G320 indicating detected infrared fluorescence areaR310 and project cutting aid line 321, which is the projection imageindicating the aid line, on the position corresponding to apredetermined condition on detected infrared fluorescence area R310.

With this, the irradiation of cutting aid line 321 can be performed inaddition to the irradiation of the area specified as affected part 140on the basis of the cutting allowable range input by the doctor prior tothe start of surgery. According to this, the reproduction of the cuttingposition planned before the start of the surgery can be assisted,whereby the burden of the doctor can be reduced. Further, the time forpreparation before the start of the surgery can be shortened.

Further, according to surgery supporting system 100, cutting aid line321 is projected according to infrared fluorescence area R310 ofaffected part 140 detected based on the emission of infraredfluorescence 310. Therefore, the doctor or the like can visuallyrecognize the aid line matching the position of affected part 140 insurgical field 135 in real time.

In surgery supporting system 100, the position on which cutting aid line321 is projected may be set to the boundary of the intensitydistribution on the basis of the intensity distribution of infraredfluorescence in the captured image.

In surgery supporting system 100, the predetermined condition may becutting allowable ranges W1 and W2 indicating the space from detectedinfrared fluorescence area R310.

Surgery supporting system 100 may further include TOF sensor 260 thatdetects distance information indicating the distance from itself toaffected part 140. Control device 230 may project cutting aid line 321at the position spaced from detected infrared fluorescence area R310 bycutting allowable ranges W1 and W2 on the basis of the distanceinformation detected by TOF sensor 260.

Notably, while cutting aid line 321 is projected at the positionuniformly away from the area specified as affected part 140 by 2centimeters when the cutting allowable range that is the predeterminedcondition is set as 2 centimeters in the above description, theconfiguration is not limited thereto. The position where cutting aidline 321 should be projected on the area specified as affected part 140may be varied according to the cutting allowable range.

Further, in the above description, the irradiation for cutting aid line321 can be turned on and off as necessary according to the operation bythe doctor or the like during the irradiation of visible laser light 320to the area specified as affected part 140. When the irradiation isturned off, the irradiation for cutting aid line 321 is not performed,and only the irradiation of visible laser light 320 to the areaspecified as affected part 140 is performed.

While the condition (cutting allowable range) for cutting aid line 321is input prior to the start of surgery in the above description, theconfiguration is not limited thereto. Specifically, the condition forcutting aid line 321 may be changeable according to the operation by thedoctor during the surgery.

7. Projection Operation of Surgery Aid Information to Surrounding ofAffected Part

7-1. Outline of Projection Operation of Surgery Aid Information

A doctor performs surgery while confirming vital data of patient 130 asnecessary. Vital data includes blood pressure, heart rate (pulse rate),oxygen concentration, and electrocardiogram. A doctor can performsurgery according to the change in condition of patient 130 byconfirming vital data. A doctor also performs surgery while confirmingan inspection image of patient 130 as necessary. Inspection imageincludes an image with MRI (Magnetic Resonance Imaging), an image withCT (Computed Tomography), and a radiographic image. A doctor can performsurgery according to the inspection result of patient 130 by confirmingthe inspection image. A doctor also performs surgery while confirming amemo indicating the procedure of the surgery or notes for the surgeryaccording to need.

As described above, a doctor performs surgery while confirming surgeryaid information such as vital data, an inspection image, and a procedureof surgery, as necessary. FIG. 12A is a view illustrating a state ofconventional surgery. Surgery aid information is displayed on monitor142. Doctor 141 performs surgery on patient 130 while confirming surgeryaid information displayed on monitor 142. At that time, doctor 141performs surgery while moving his/her eye to monitor 142 and patient130, which increases a burden of doctor 141 and increases time forconfirmation.

In view of this, the inventor of the present disclosure has conceived ofprojecting surgery aid information 151 around affected part 140 inaddition to the projection of a visible-light image onto an areaspecified as affected part 140. With this, the eye movement of a doctoror the like can be reduced during surgery. Thus, the burden on thedoctor or the like can be reduced, and the confirmation time can bereduced.

7-2. Detail of Projection Operation of Surgery Aid Information

The projection of surgery aid information onto the surrounding ofaffected part 140 will be described with reference to FIGS. 12B, 13A,and 13B. FIG. 12B is a view for describing projection of surgery aidinformation 151 onto the surrounding of affected part 140. FIGS. 13A and13B are views for describing the projection of surgery aid information151 onto an auxiliary screen material 150.

Control device 230 in surgery supporting system 100 is communicativelyconnected to a medical device (not illustrated) from which various vitaldata is acquired. With this, control device 230 acquires vital datarequired for surgery in real time from the communicatively-connectedmedical device.

Further, inspection image data of patient 130 and a memo of theprocedure of the surgery are previously stored in memory 240 through theoperation on the operation unit by doctor 141 prior to the start of thesurgery. Thus, control device 230 reads and acquires the inspectionimage data and the memo of the procedure of the surgery necessary forthe surgery from memory 240.

FIG. 12B is a view illustrating the state of the projection of surgeryaid information 151 according to the present exemplary embodiment. Priorto the start of the surgery, doctor 141 or the like places auxiliaryscreen material 150 on which surgery aid information 151 is projectedaround affected part 140 of patient 130 as illustrated in FIG. 12B. Anymaterial may be used for auxiliary screen material 150, so long as itcan display a projection image. Further, a material having any shape andsize may be used for auxiliary screen material 150, so long as it has asize placeable around affected part 140. In the example in FIG. 12B,auxiliary screen material 150 is placed at the right of affected part140 viewed from doctor 141. However, the placing position is not limitedthereto. Auxiliary screen material 150 may be placed at any positionaround affected part 140 according to the dominant arm of the doctorusing surgery supporting system 100, easiness in confirmation, or thesurgical matter.

FIG. 13A is a top view of auxiliary screen material 150 on which surgeryaid information is not projected. As illustrated in FIG. 13A, markers152 are formed on the top surface of auxiliary screen material 150.Markers 152 are positioned on auxiliary screen material 150 as areference indicating the area on which surgery aid information 151 isdisplayed on auxiliary screen material 150.

A camera (not illustrated) is connected to control device 230 of surgerysupporting system 100. The camera captures an image of markers 152formed on auxiliary screen material 150. The camera transmits thecaptured image of markers 152 to control device 230. The correspondencerelation in coordinates between the captured area in the captured imageby the camera and the projection area of surgery aid information byvisible-light laser 222 is stored in memory 240 in advance. Controldevice 230 specifies the area on which surgery aid information 151 is tobe projected from the correspondence relation stored in memory 240 andthe detection result of the positions of markers 152 from thetransmitted captured image. Then, control device 230 controls MEMSmirror 221 such that surgery aid information 151 is projected onto thespecified area. Thus, projection image G151 indicating surgery aidinformation 151 is projected on the top surface of auxiliary screenmaterial 150 as illustrated in FIG. 13B.

Surgery supporting system 100 projects projection image G151 of surgeryaid information 151 on auxiliary screen material 150 along withprojection image G320 (see FIG. 2B) projected on infrared fluorescencearea R310 specified as affected part 140. With this, the eye movement ofthe doctor can be reduced during the surgery. Thus, the burden on doctor141 can be reduced, and the confirmation time can be reduced, wherebythe surgery can be supported.

While surgery aid information 151 is projected on auxiliary screenmaterial 150 in the above description, it is not limited thereto.Surgery aid information 151 may be directly projected on the surface ofthe body of patient 130, not on auxiliary screen material 150. In thiscase, markers 152 may be formed on the surface of the body of patient130.

While the camera for capturing an image of markers 152 is used in theabove description, the configuration is not limited thereto. Forexample, an image of markers 152 may be captured by infrared camera 210.In this case, markers 152 are made of a material formed by applying ICGthereon, kneading ICG therein, or injecting ICG therein, for example.With this, images of affected part 140 and markers 152 can be capturedonly by infrared camera 210.

While the area on which surgery aid information 151 is to be projectedis specified by using markers 152 in the above description, theconfiguration is not limited thereto. Specifically, the area on whichsurgery aid information 151 is to be projected may be specified withoutusing markers 152. For example, surgery aid information 151 may beprojected at the position away, by the distance in the direction as setby the doctor beforehand, from the position of affected part 140 towhich visible laser light 320 is emitted.

For example, it is supposed to be set beforehand such that surgery aidinformation 151 is projected at the position away from the rightmost endof the area specified as affected part 140 to the right by 20centimeters as viewed from doctor 141. In this case, control device 230controls MEMS mirror 221 such that surgery aid information 151 isprojected on the position set beforehand with respect to the areaspecified as affected part 140. Thus, surgery aid information 151 can beprojected on an arbitrary position which is easy to be confirmed bydoctor 141. It is to be noted that control device 230 may calculate theposition on which surgery aid information 151 is to be projected insurgical field 135 on the basis of distance information detected by TOFsensor 260.

7-3. Effects, Etc.

As described above, in the present exemplary embodiment, surgerysupporting system 100 includes infrared camera 210, projector 220, andcontrol device 230. Infrared camera 210 captures an image of affectedpart 140. Projector 220 generates projection image G320 by visible lightand projects the resultant on affected part 140. Control device 230controls the projection operation of projector 220 on the basis of thecaptured image captured by infrared camera 210. Control device 230controls projector 220 such that projection image G320 indicatingcaptured affected part 140 is projected and projection image G151indicating surgery aid information 151, which is the informationconcerning the surgery to affected part 140, is projected in thevicinity of affected part 140.

Thus, projection image G151 is projected in the vicinity of affectedpart 140, whereby an eye movement of a doctor or the like from affectedpart 140 when a doctor or the like confirms surgery aid information 151can be decreased to reduce a burden on the doctor during the surgery.

Further, surgery supporting system 100 may further include auxiliaryscreen material 150 that is disposed near affected part 140 and hasmarkers 152. In this case, control device 230 projects projection imageG151 on auxiliary screen material 150 by using the positions of markers152 as a reference. Surgery supporting system 100 may further include acamera for capturing an image of markers 152, or may capture an image ofmarkers 152 by infrared camera 210.

Surgery supporting system 100 may further include memory 240 that storessurgery aid information 151. Further, control device 230 may acquiresurgery aid information 151 through communication with an externaldevice.

Surgery supporting system 100 may further include a distance detectionunit such as TOF sensor 260 that detects distance information indicatingthe distance from itself to affected part 140. Control device 230 maycause projector 220 to project surgery aid information 151 at theposition away from affected part 140 by a predetermined distance on thebasis of the detected distance information. Control device 230 may alsocause projector 220 to project surgery aid information 151 on almost aflat area near affected part 140 on the basis of the detected distanceinformation. The distance detection unit may output a distance image asdistance information, for example.

8. Monitoring of Use Height of Imaging Irradiation Device

8-1. Outline of Monitoring Operation of Use Height

In surgery supporting system 100 illustrated in FIG. 1, use heights ofimaging irradiation device 200 and surgical bed 110 are adjusted at thestart of surgery such that the body axis of patient 130 is located atthe position 1000 mm away from the lower surface of imaging irradiationdevice 200 according to the height allowable range of 1000 mm±300 mmbased on the focal length of infrared camera 210, for example. However,during the surgery, patient 130 may be turned over according to thematter of the surgery or the location of imaging irradiation device 200may be changed due to a change in operators, so that the use heights ofimaging irradiation device 200 and surgical bed 110 are changed.

According to the present invention, a distance detection unit isprovided to imaging irradiation device 200 so as to monitor the useheight of imaging irradiation device 200 during surgery, whereby patient130 can be turned over or the height of the surgical bed can be adjustedaccording to the matter of the surgery within an allowable range of theuse height. On the other hand, when the use height is outside theallowable range, a warning is issued to avoid false recognition of auser such as a doctor. Further, safety during surgery can be ensuredunder the control such that a projection image is not projected when theuse height is outside the allowable range.

In the present exemplary embodiment, TOF sensor 260 that radiatesinfrared detection light 330 with wavelength of 850 nm to 950 nm is usedas the distance detection unit. Infrared detection light 330 radiatedfrom TOF sensor 260 is reflected on the surface of the body of patient130, and then, returns to TOF sensor 260 to be received. In this case,infrared detection light 330 reflected on the surface of the body ofpatient 130 reaches not only TOF sensor 260 but also infrared camera210.

The present invention includes a configuration for controlling TOFsensor 260 and infrared excitation light source 250 or visible-lightlaser 222 in an opposite way in order to monitor a safe use height whileimplementing a surgery support with the detection of infraredfluorescence 310.

8-2. Detail of Monitoring Operation of Use Height

The detail of the monitoring operation of a use height will be describedbelow.

8-2-1. With Regard to Process Flow

Firstly, the process flow in the monitoring operation of the use heightin surgery supporting system 100 will be described with reference toFIGS. 14, 15A, and 15B. FIG. 14 is a flowchart illustrating the processflow in the monitoring operation of a use height. FIGS. 15A and 15B areviews for describing the monitoring operation of a use height. This flowis executed by control device 230 in surgery supporting system 100 (seeFIG. 1).

In the flowchart in FIG. 14, TOF sensor 260 firstly radiates infrareddetection light 330 and receives its reflection wave to detect distancedi from itself to patient 130 under the control of control device 230(S200) as illustrated in FIG. 15A. In step S200, TOF sensor 260 radiatesinfrared detection light 330 only during predetermined period T1 (seeFIG. 16). TOF sensor 260 outputs the detected distance information tocontrol device 230.

Then, control device 230 determines whether or not detected distance difalls within predetermined first segment r1 on the basis of the distanceinformation from TOF sensor 260 (S202). First segment r1 indicates theallowable range of the distance between imaging irradiation device 200and affected part 140 by which surgery supporting system 100 is normallyoperable. In the present exemplary embodiment, first segment r1 is setto be 1000±300 mm with d0 being specified as a standard distance that is1000 mm.

When determining that detected distance di is di′, and di′ is outsidefirst segment r1 as illustrated in FIG. 15B (NO in S202), control device230 issues a warning indicating that the use height is abnormal (S214).The warning in step S214 may be generated such that a message or awarning sound indicating that the use height is in the “abnormal state”is issued from a speaker (not illustrated). Notably, in step S214, aprojection image is not projected on affected part 140 as illustrated inFIG. 15B.

Then, after the lapse of period T1, control device 230 causes TOF sensor260 to detect distance di from itself to patient 130 as in the processin step S200 (S216).

Then, control device 230 determines whether or not detected distance difalls within predetermined second segment r2 (S218). Second segment r2indicates that surgery supporting system 100 is located at the positionwhere it can be returned from the abnormal state. Second segment r2 isshorter than first segment r1, and r2 is 1000±200 mm, for example.

When determining that detected distance di is outside second segment r2(NO in S218), control device 230 repeatedly performs the process in stepS216 in a predetermined cycle. On the other hand, when determining thatdetected distance di falls within second segment r2 (YES in S218),control device 230 sequentially performs the processes in step S204 andsubsequent steps.

When determining that detected distance di falls within first segment r1as illustrated in FIG. 15A (YES in S202), control device 230 controlsinfrared excitation light source 250 (see FIG. 1) such that infraredexcitation light 300 is emitted to surgical field 135 (S204).

During the irradiation of infrared excitation light 300, control device230 causes infrared camera 210 to capture affected part 140 in surgicalfield 135 (S206). Control device 230 causes projector 220 to projectprojection image G320 of visible light on the basis of the imagecaptured in the process of step S206 (S208). The processes in stepsS202, S204, and S206 are performed similarly to the above-mentionedbasic projection operation in surgery supporting system 100 (see FIG.2).

Then, control device 230 determines whether or not predetermined periodT2 has elapsed after the start of the irradiation of infrared excitationlight 300 in step S204 (S210). Control device 230 repeatedly executesthe processes in steps S206 and S208 in a predetermined cycle (e.g.,1/60 second) until period T2 has elapsed (NO in S210). After the lapseof period T2 (YES in S210), control device 230 causes infraredexcitation light source 250 to stop the irradiation of infraredexcitation light 300, and causes projector 220 to erase projection imageG320 (S212). After the process in step S212, control device 230 returnsto the process in step S200.

It is to be noted that, since TOF sensor 260 detects the distance usinginfrared light, control device 230 stops other light sources in stepS212 before returning to step S200 to prevent influence on the distancedetection. However, if projector 220 has a configuration of using alight source having no infrared light component, which configuration hasno influence on the distance detection, control device 230 may onlycause infrared excitation light source 250 to stop the irradiation ofinfrared excitation light 300 in step S212.

According to the above processes, a warning is generated in the processin S214 when the use height of imaging irradiation device 200 is outsidethe allowable range, whereby a user such as a doctor can recognize thatthe use height of imaging irradiation device 200 is outside theallowable range. Further, the distance detection process in step S200 isperformed after the process in step S212 to control TOF sensor 260 andprojector 220 or the like in an opposite way, whereby the distancedetection can be implemented without causing malfunction of surgerysupporting system 100.

8-2-2. With Regard to Opposite Control

The opposite control in monitoring the use height of imaging irradiationdevice 200 will be described below in detail with reference to FIGS. 14to 16. FIG. 16 is a timing chart for describing the operation ofinfrared excitation light source 250, TOF sensor 260, and visible-lightlaser 222 according to a height detection result. The horizontal axis inFIG. 16 indicates a time axis. In FIG. 16, a low level in each chartindicates an unlit state, while a high level indicates a lit state.

Here, the “lit state” indicates that a power source of each of infraredexcitation light source 250, TOF sensor 260, and visible-light laser 222is turned on. On the other hand, the “unlit state” indicates that thepower source of each of infrared excitation light source 250, TOF sensor260, and visible-light laser 222 is turned off.

Control device 230 periodically executes the height (distance)determination by TOF sensor 260 during the operation of surgerysupporting system 100. Specifically, as illustrated in FIG. 16, controldevice 230 executes the determination process in steps S200 and S202 orin steps S216 and S218 in FIG. 14 during period T1 from times t1 to t2,during period T1 from times t3 to t4, during period T1 from times t5 tot6, during period T1 from times t7 to t8, during period T1 from times t9to t10, . . . . In this case, TOF sensor 260 is in a lit state ofradiating infrared detection light 330. Control device 230 causesinfrared excitation light source 250 and visible-light laser 222 to bean unlit state during each period T1 in which TOF sensor 260 performsheight determination. That is, the opposite control for bringinginfrared excitation light source 250 and visible-light laser 222 intothe unlit state during the lit state of TOF sensor 260 is performed.Period T1 from times t1 to t2, period T1 from times t3 to t4, period T1from times t5 to t6, period T1 from times t7 to t8, and period T1 fromtimes t9 to t10 are all short periods such as 10 msec to 100 msec.Therefore, these periods are hardly sensed by a human. Therefore, evenif the opposite control is performed during the period of heightdetermination, surgery supporting system 100 can make a human feel as ifthe projection image by visible laser light 320 is continuouslydisplayed.

It is supposed here that, during period T1 from times t1 to t2, distancedi indicated by the detection result of TOF sensor 260 falls withinfirst segment r1 (=1000 mm±300 mm) indicating the height allowable rangeas illustrated in FIG. 15A. In this case, control device 230 determinesthat the use height is in the “normal state” as the result of the heightdetermination in step S202 in FIG. 14. Then, control device 230 bringsboth infrared excitation light source 250 and visible-light laser 222into the lit state during subsequent period T2 from times t2 to t3.Thus, during period T2 from times t2 to t3, projection image G320 isprojected onto affected part 140 to normally support the surgery asusual.

Next, it is supposed that, during period T1 from times t3 to t4,distance di indicated by the detection result of TOF sensor 260 is di′which is outside height allowable range r1 (=1000 mm±300 mm) asillustrated in FIG. 15B. In this case, control device 230 determinesthat the use height is in the “abnormal state” as the result of theheight determination in step S202. In this case, the projection image islikely to be inaccurately projected, so that it is considered that ithad better not to continue the surgery support from the viewpoint ofsafety. Then, control device 230 keeps both infrared excitation lightsource 250 and visible-light laser 222 into the unlit state even duringsubsequent period T2 from times t4 to t5. Thus, during period T2 fromtimes t4 to t5, a projection image which may be inaccurate is notdisplayed, and the surgery support can be stopped with safety beingprioritized.

It is supposed that, during subsequent period T1 from times t5 to t6,distance di indicated by the detection result of TOF sensor 260 isbetween distance d1 at one end of first segment r1 and distance d2 atone end of second segment r2 (e.g., 1250 mm). In this case, imagingirradiation device 200 falls within the height allowable range, butsince the use height is close to the limit of the height allowablerange, there is a fear that the use height immediately falls outside theheight allowable range. In view of this, a hysteresis range is formed insecond segment r2 smaller than first segment r1, and the determinationprocess in step S218 in FIG. 14 is performed. With this, when distancedi is between first distance d1 and distance d2, it is determined to bein the “abnormal state”, so that the operation similar to that duringperiod T2 from times t4 to t5 is performed to ensure safety even duringperiod T2 from times t6 to t7.

Next, it is supposed that, during period T1 from times t7 to t8,distance di indicated by the detection result of TOF sensor 260 fallswithin second segment r2 (=1000 mm±200 mm) indicating the heightallowable range with the hysteresis range. In this case, control device230 determines that the use height is in the “normal state” as theresult of the height determination in step S218. Then, control device230 brings both infrared excitation light source 250 and visible-lightlaser 222 into the lit state during subsequent period T2 from times t8to t9. Thus, during period T2 from times t8 to t9, projection image G320is projected onto affected part 140 as normal, and the surgery supportcan be performed again.

Notably, a margin period in which TOF sensor 260 as well as infraredexcitation light source 250 and visible-light laser 222 are in the unlitstate may be formed at the switching timing of t1, t3, t5, t7, t9, . . .. . With this, erroneous detection of infrared detection light 330 atthe switching timing can be prevented. In addition, a margin period maybe formed at the switching timing of t2, t4, t6, t8, t10, . . . .

8-3. Effects, Etc.

As described above, in the present embodiment, surgery supporting system100 includes infrared camera 210, projector 220, TOF sensor 260, andcontrol device 230. Infrared camera 210 captures an image of affectedpart 140. Projector 220 generates projection image G320 on the basis ofthe captured image of affected part 140 and projects the resultant imageonto affected part 140. TOF sensor 260 detects the distance from itselfto affected part 140. Control device 230 controls the operations ofinfrared camera 210 and projector 220. Control device 230 determineswhether or not the distance detected by TOF sensor 260 falls withinfirst segment r1. When the distance detected by TOF sensor 260 fallswithin first segment r1, control device 230 generates projection imageG320 and projects this image onto affected part 140.

According to surgery supporting system 100 described above, when thedistance detected by TOF sensor 260 falls within first segment r1,projection image G320 is generated and projected on affected part 140even if the imaging position of affected part 140 is changed. Thus,safety in projecting projection image G320 can be ensured.

Further, control device 230 generates a predetermined warning when thedistance detected by TOF sensor 260 is outside first segment r1.

Thus, surgery supporting system 100 teaches a doctor or the like thatthe distance from TOF sensor 260 to affected part 140 exceeds firstsegment r1, and thus is capable of ensuring safety while in use.Accordingly, surgery supporting system 100 can be easy to be used by auser such as a doctor.

Further, when the distance detected by TOF sensor 260 is outside firstsegment r1, projection image G320 may not be projected on affected part140 in place of or in addition to the configuration of generating apredetermined warning. With this, the projection of projection imageG320 which may be inaccurate because of the detected distance exceedingfirst segment r1 is stopped in surgery supporting system 100, wherebysafety during surgery can be enhanced.

In addition, infrared camera 210 may receive infrared fluorescence 310having a first spectrum and capture an image of affected part 140. TOFsensor 260 may radiate infrared detection light 330 having a secondspectrum to detect the distance from itself to affected part 140. Inthis case, TOF sensor 260 radiates infrared detection light 330 duringfirst period T1, and does not radiate infrared detection light 330during second period T2 different from first period T1. Control device230 does not allow the projection of projection image G320 during firstperiod T1 and allows the projection of projection image G320 duringsecond period T2. Thus, the distance from TOF sensor 260 to affectedpart 140 can be detected without causing malfunction of surgerysupporting system 100.

In the above description, infrared light source 250 and visible-lightlaser 222 are both brought into the unlit state when the determinationof the “abnormal state” is made. However, the configuration is notlimited thereto. One of infrared excitation light source 250 andvisible-light laser 222 may be turned off. When infrared excitationlight source 250 is brought into the unlit state, infrared fluorescence310 is not emitted from ICG. Therefore, control device 230 cannotspecify an area of affected part 140, so that visible laser light 320 isnot radiated even if visible-light laser 222 is in the lit state.Further, when visible-light laser 222 is brought into the unlit state,visible laser light 320 is never emitted.

In addition, when the determination of the “abnormal state” is made,control device 230 may cause infrared camera 210 not to capture an imagein place of or in addition to the control of infrared excitation lightsource 250 and visible-light laser 222. Further, control device 230 maycause MEMS mirror 221 not to generate projection image G320. That is,control device 230 may control any of components in surgery supportingsystem 100 in order that projection image G320 based on the capturingresult of infrared camera 210 is not projected.

Further, the example of the warning operation for issuing, from aspeaker, a message or a warning sound indicating that the use height isin the “abnormal state” when the determination of the “abnormal state”is made has been described above. However, the warning operation is notlimited thereto. The warning may be an operation for outputtinginformation indicating that the distance to a subject such as affectedpart 140 is outside a predetermined segment range. For example, when theuse height is determined to be in the “abnormal state”, a user such as adoctor may be notified that it is in the “abnormal state”. The methodfor notifying the “abnormal state” may be such that visible-light laser222 is changed to the one having other wavelength, and visible laserlight 320 may be emitted with its color being changed. Further, awarning may be issued by projecting a projection image including a textmessage that indicates the “abnormal state”.

In the above description, projection image G320 based on the capturingresult of infrared camera 210 is erased upon generating a warning.However, the configuration is not limited thereto. For example, aprojection image based on the capturing result of infrared camera 210may be used as a warning. For example, a projection image based on thecapturing result of infrared camera 210 may be projected with its colorbeing changed or may be projected while flashing.

In the above description, TOF sensor 260 uses infrared detection light330 with a spectrum superimposed on the spectrum of infraredfluorescence 310. However, it is not limited thereto. For example, TOFsensor 260 may radiate detection light not superimposed on the spectrumof infrared fluorescence 310 to perform distance detection. In thiscase, a wavelength filter that blocks the spectrum of infraredfluorescence 310 may be provided on a light-emitting unit of TOF sensor260, for example. Here, a wavelength band with wavelength of 850 nm to950 nm has high atmospheric transmittance, so that the distancedetection is easy to be performed. Therefore, the efficiency in thedistance detection can be enhanced by performing the opposite controldescribed above without using a wavelength filter.

In the above description, a use height is monitored with the distancedetection by TOF sensor 260. However, it is not limited thereto. Forexample, the distance between imaging irradiation device 200 and asubject such as affected part 140 may be monitored with the distancedetection by TOF sensor 260 in order that surgery supporting system 100can appropriately operate even when the direction of imaging irradiationdevice 200 is changed.

Further, it has been described with reference to FIG. 16 that the statetransition of each of infrared excitation light source 250 andvisible-light laser 222 between the “lit state” and the “unlit state” isimplemented by switching the power source of the light source between anon state and an off state. However, the configuration is not limitedthereto. The “lit state” and the “unlit state” may be implemented byswitching an on state and an off state for light-shielding by alight-shielding unit, even if the power source of the light source iskept into the on state.

Other Exemplary Embodiments

The first exemplary embodiment has been described above as anillustration of the technology disclosed in the present application.However, the technology in the present disclosure is not limited tothis, and can be applied to embodiments in which various changes,replacements, additions, omissions, etc., are made as necessary.Furthermore, a new embodiment can be formed by combining the componentsdescribed in the first exemplary embodiment.

The other exemplary embodiments will be described below.

The first exemplary embodiment describes as one example a medicalapplication such as surgery. However, the present invention is notlimited thereto. For example, the present invention is applicable to thecase where work is performed on a subject of which state change cannotbe confirmed by visual observation in, for example, a construction site,a mining site, a building site, or a material processing plant.

Specifically, in place of the medical device according to the firstexemplary embodiment, a fluorescent material may be applied on, kneadedinto, or injected into a subject of which state change cannot beconfirmed by visual observation, and the resultant subject may be usedas a target to be captured by infrared camera 210, in a constructionsite, a mining site, a building site, or in a material processing plant,for example. Not a portion emitting light but a portion generating heatmay be detected by a heat sensor, and only this portion or only theboundary may be scanned.

The first exemplary embodiment describes using a laser light source.However, the configuration is not limited thereto for projection of acutting aid line or projection of surgery aid information. That is, acutting aid line or surgery aid information may be projected using alight source other than a laser light source.

The first exemplary embodiment describes that a cutting aid line orsurgery aid information is projected using visible-light laser 222 thatis the same as a light source used for the projection to an areaspecified as an affected part. However, the configuration is not limitedthereto. The cutting aid line or surgery aid information may beprojected using a light source different from the light source used forthe projection to the area specified as the affected part. However, itis obvious that such light source is controlled to perform projectioncorresponding to the projection to the area specified as the affectedpart.

In the first exemplary embodiment, TOF sensor 260 is used as thedistance detection unit. However, it is not limited thereto. Forexample, a sensor may be used that radiates infrared detection lighthaving a known pattern such as a random dot pattern and measuresdistance on the basis of the shift with the pattern of its reflectionwave. In this case, the distance detection unit can detect distanceinformation as a distance image indicating distance to each dot in atwo-dimensional area.

Further, the first exemplary embodiment describes projection image G320of a monochrome uniform image by visible-light laser 222. The projectionimage projected by the projector is not limited thereto, and agray-scaled projection image or a full-color projection image may beprojected, or an arbitrary image may be projected.

As presented above, the other exemplary embodiments have been describedas an example of the technology according to the present disclosure. Forthis purpose, the accompanying drawings and the detailed description areprovided.

Therefore, components in the accompanying drawings and the detaildescription may include not only components essential for solvingproblems, but also components that are provided to illustrate the abovedescribed technology and are not essential for solving problems.Therefore, such inessential components should not be readily construedas being essential based on the fact that such inessential componentsare shown in the accompanying drawings or mentioned in the detaileddescription.

Further, the above described exemplary embodiments have been describedto exemplify the technology according to the present disclosure, andtherefore, various modifications, replacements, additions, and omissionsmay be made within the scope of the claims and the scope of theequivalents thereof.

A projection system according to the present disclosure is applicable tomedical application and to the case where work is performed on a subjectof which state change cannot be confirmed by visual observation in, forexample, a construction site, a mining site, a building site, or amaterial processing plant.

What is claimed is:
 1. A projection system comprising: a box-likehousing including a surface with a reference area; a light sourceenclosed within the box-like housing, the light source being configuredto emit light including non-visible light and visible light through thereference area; an imaging unit that receives the non-visible light andcaptures an image of the non-visible light emitted through the referencearea; and a projector that projects a projection image by visible lighton the surface of the box-like housing through which the non-visiblelight is emitted based on the captured image of the non-visible lightemitted through the reference area, wherein the box-like housing isplaced where a patient's body is placed, and the box-like housing isplaced, instead of the patient's body, before the projector projects theimage onto the patient's body.
 2. The projection system according toclaim 1, further comprising: an adjustment unit that compares thereference area with the projection image on the surface and adjusts aposition of the projection image based on a result of the comparison. 3.The projection system according to claim 2, wherein the adjustment unitdetermines whether or not the reference area and the projection imageare at least partly overlapped with each other on the surface, and whenthe reference area and the projection image are at least partlyoverlapped with each other, the adjustment unit does not adjust theposition of the projection image.
 4. The projection system according toclaim 2, further comprising a storage unit, wherein the adjustment unitcalculates, based on the result of the comparison, a correction amountfor aligning the reference area with the projection image, and storesthe calculated correction amount into the storage unit.
 5. Theprojection system according to claim 4, wherein the correction amount isinformation for correcting at least one of a position, an angle, aprojection magnification, and distortion of the projection imageprojected by the projector.
 6. The projection system according to claim1, wherein the non-visible light includes infrared light.
 7. Theprojection system according to claim 1, further comprising alight-shielding member that shields light emitted from the light sourceand has an opening portion corresponding to the reference area.
 8. Theprojection system according to claim 1, wherein the surface furtherincludes a screen material.
 9. The projection system according to claim2, wherein the surface further includes a screen material.
 10. Theprojection system according to claim 3, wherein the surface furtherincludes a screen material.
 11. The projection system according to claim4, wherein the surface further includes a screen material.
 12. Theprojection system according to claim 5, wherein the surface furtherincludes a screen material.
 13. The projection system according to claim6, wherein the surface further includes a screen material.
 14. Theprojection system according to claim 7, wherein the surface furtherincludes a screen material.
 15. The projection system according to claim1, wherein the surface is disposed on the box-like housing.